TO JET: The Turkish Online Journal of Educational Technology - January 2017, volume 16 issue 1 



Reading from Multimedia Materials: Benefits of Non-congruent Pictures on Reading 
Comprehension for Dyslexic Readers 


Eva Wennas Brante 

Department of Pedagogical Curricular and Professional Studies, Gothenburg University, Sweden 
eva. wennas. brante@gu.se 


Mona Holmqvist 

Faculty of Education and Society, Malmo University, Sweden 
mona.holmqvist@mah.se 


ABSTRACT 

Pictures are often integrated in digital learning materials with the purpose of enhancing learning. This mixed 
methods study uses quantitative eye-tracking data and qualitative data such as oral answers to discover whether 
characteristics of pictures influence patterns of text-picture transition in readers with (n= 10) and without («= 14) 
dyslexia, and how reading comprehension is affected. Most participants attended to the picture with a “non- 
congruent with reality” motif early in the inspection process. Qualitative analysis of oral answers showed that 
retaining the gist of that specific picture led to more developed answers, even for the dyslexic group. Early 
attention to the picture thus gives readers a fair chance of starting with a holistic impression of the material to be 
processed. 
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INTRODUCTION 

Pictures are often used in digital learning materials with the purpose of enhancing learning, unfortunately 
without accounting for readers’ different cognitive abilities (Scheiter, Wiebe, & Holsanova, 2008). For example, 
Authors (Submitted) found in an eye-tracking study that participants with dyslexia inspected pictures later than 
those without dyslexia. This general tendency, however, was drawn from an analysis that did not distinguish 
between different pictures. Analyzed separately, would different pictures result in different transition patterns for 
the two groups? Our aim of this paper is to investigate how and whether inspection patterns of three different 
pictures affect reading comprehension for dyslexic and non-dyslexic participants. To achieve this aim we have 
made use of a mixed methods design, integrating quantitative eye-tracking data such as fixations and dwell times 
with a qualitative analysis of other visual (quantitative) data such as scan paths, but also oral answers from the 
participants in a convergent parallel design (Klassen, Creswell, Clark, Smith, & Meissner, 2012). As the use of 
eye-tracking measurements is related to the eye-mind hypothesis of Just and Carpenter (1980), which states that 
where individuals place their gaze they also place their attention, we argue that merely quantitative data cannot 
account for the complexity of processes running when readers encounter materials consisting of both text and 
pictures. Frequently, research in this field employs a design that involves first analyzing the quantitative eye¬ 
tracking data to establish visual patterns, then adding scores from reading comprehension questions to explain 
the learning effect. Still, this leaves the question of why participants choose to inspect the material in a specific 
way unresolved. The focus in our study is to integrate quantitative and qualitative data, collected at the same 
time, to better understand how two groups of participants handle material where pictures are integrated into texts 
and their reading comprehension evolves. Figure 1 illustrates how each data set contributes to the research 
question (shortened here). 
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QUAN analysis of 
eye-tracking 
shows groups' 
inspection 
patterns 


QJJAl analysis of 
QUAN scan paths 
shows when 
pictures are 
attended to 


How do inspection 
patterns of 
pictures and texts 
affect reading 
comprehension? 


QUAL analysis of 
oral answers 
shows reading 
comprehension 


Figure 1 . How the data sets are related to the research question 


Before we elaborate more on our study, we would like to return to the issue of pictures in texts and what is 
known from earlier research in the field, although it is important to bear in mind that this research has not 
focused on dyslexic readers. Previous studies have found that when presented with learning materials in an 
uncontrolled order, most people tend to start with reading and then look at the picture (Hegarty & Just, 1993; 
Rayner, Rotello, Stewart, Keir, & Duffy, 2001; Schmidt-Weigand, Kohnert, & Glowalla, 2010). For example, in 
a well-known study by Hegarty and Just (1993), participants studied diagrams of pulleys and read texts about 
them. Results showed a strong tendency to read first and then to inspect the schematic picture. Rayner, Rotello et 
al. (2001) showed in another study that most people read the text in an advertisement before inspecting the 
picture connected to it, even if some did make an initial inspection of the picture (Rayner et al., 2001). Schmidt- 
Weigand, Kohnert and Glowalla (2010) let students read about lightning in different experimental conditions; 
those who were presented with both text and picture started to read the text before inspecting the pictures. That 
study also showed that when participants had no time restriction in the experiment, they spent a longer time 
reading the text, but visual inspection of the illustrations did not increase (Schmidt-Weigand et al., 2010). The 
shorter time dedicated to inspecting pictures in all studies could be explained by a cultural impulse to ascribe 
written words more importance than pictures (Jewitt, 2015). Some studies have explicitly investigated how 
pictures influence reading comprehension for either readers with dyslexia or readers characterized as weak 
readers. One example is a study by Beacham and Alty (2006) with only dyslexic participants, inspecting learning 
material consisting of text and picture or only text. The best learning outcomes were found for the group who 
were given only text (Beacham & Alty, 2006). In contrast to this, Houts, Doak, Doak, & Loscalzo (2006) found 
positive learning effects of pictures being presented with text for individuals with low literacy. A later study by 
SaB and Schiitte (2015) showed that pictures and texts must contain referential connections to be of use for weak 
readers. Obviously, more research regarding how readers with dyslexia handle multimedia is needed. 

In an experiment with controlled conditions (non-dyslexic participants), positive learning effects were obtained 
by digitally showing a schematic picture some milliseconds before a text about the same phenomenon is read 
aloud (Eitel, Scheiter, Schuler, Nystrom, & Holmqvist, 2012). Eitel et al. (2012) exposed participants to a 
schematic picture of a pulley system (for 600 ms, 2 s or self-paced) prior to their hearing a text about the pulley’s 
function. The 600 ms inspection provided spatial information on a global level; in the self-paced condition it was 
also possible to perceive more detailed information on a local level, which some participants did. The results of 
this experiment suggest that learners start with a holistic understanding of a picture and are later able to allocate 
meaning to various parts of the picture. This is in accordance with the reasoning in variation theory about how 
learning takes place (Marton & Booth, 1997). Marton and Booth write that “learning proceeds, as a rule, from an 
undifferentiated and poorly integrated understanding of the whole to an increased differentiation and integration 
of the whole and its parts” (1997: viii). But without knowledge, however fragmentary, of the whole that a learner 
is dealing with, its parts will not be meaningful. A fragmentary knowledge of the whole may be comparable to 
the gist of a picture, perceived within a few milliseconds. 

The gist - a holistic start to processing visual material 

The gist of a picture is a very rudimentary impression that can be formulated as a word or a short phrase such as 
“it’s a garden” or “street” (Underwood, 2005). A person can perceive a scene on a global level and have a first 
brief impression that will later help on a local level to identify details or objects congruent with that setting. 
Underwood (2005) claims, after going through studies exploring the significance of the gist, that acquiring the 
gist is “a case of holistic processing facilitating local processing” (p.169). From that perspective, the gist is 
crucial when viewing pictures, especially when they are integrated with texts. Underwood and Foulsham (2006) 
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found that if a picture included an object in an obviously wrong place (e.g. a vacuum cleaner on a lawn), that 
object attracted fixations at an early stage of processing the picture; the gist of the picture was violated when the 
objects in it were not congruent with the setting. It is as a result of this line of reasoning that we have used the 
term “non-congruenf ’ in this article when characterizing the surrealistic picture (see Figure 5) and distinguishing 
it in particular from the other two pictures. The surrealistic picture shows objects not congruent with the setting. 
Getting the gist of a picture has also been shown to support making correct judgments about sentences shown 
after the exposure (Calvo, Nummenmaa, & Hyona, 2008), thus the gist can contribute to reading comprehension. 
This could be especially important for readers with dyslexia, who still have to use basic cognitive resources such 
as decoding, and who tend not to further inspect pictures as early as those without dyslexia. 

Dyslexia and reading 

Dyslexia is commonly explained as a syndrome with weak phonological coding as an underlying factor 
(Guardiola, 2001; Laasonen, Lipsanen, & Virsu, 2012; Paulesu et al., 2001; Snowling, 2000). As a syndrome, 
dyslexia can manifest in various ways in different individuals (Laasonen et al., 2012). In spite of the 
heterogeneity, phonological processing deficits (Peterson & Pennington, 2015) constitute a common feature, 
which is critical in the process of fluent reading (Everatt & Reid, 2009). The deficits in phonological ability 
result in problems mapping sounds to signs and vice versa, which in turn aggravate word decoding and slow 
down reading fluency (Engen & Hoien, 2002). Reading is more demanding for dyslexic readers and research 
shows disparate findings with regard to whether pictures embedded in texts are helpful or not for the reading 
comprehension of dyslexic readers. 

THE CURRENT STUDY 

The overarching question explored in this paper is how pictures affect reading comprehension; consequently the 
visual data is placed in focus, treated both quantitatively and qualitatively. Bazeley and Kemp (2012) stated that 
“just having different sources of data does not imply mixing of methods” (p. 69). Despite that, it seems to be 
common, according to Greene (2008), that studies only mix data sets in the discussion section when inferences 
about the results are made, instead of the more challenging approach of mixing data types in the analysis. Mixing 
in the analysis could, for example, consist of transforming data from quantitative to qualitative or comparing one 
set of data with another data set during the analysis process, thus letting one data type inform another (Greene, 
2008). The earlier in the paper the integration takes place the better (Bazeley & Kemp, 2012). A promising 
integration was found in a small-scale case study by Levesque and Corrigan (2014), who analyzed eye-tracking 
data and think-aloud protocols from three participants’ inspections of online photographs in the domain of 
history. In the discussion section, they consistently integrated results from the two databases, drawing inferences 
such as, for example, showing that younger, more media-familiar students both benefit and do not benefit from 
online visual material. Without the mixed methods approach the conclusions would have leaned in one direction, 
failing to detect the divergence in the results. Against this backdrop we would like to assert that integrating eye¬ 
tracking movements with qualitative data sets is a successful way to achieve the complex answers that complex 
questions call for. 

Research must considerate the effect of dyslexia on the ability to process multimodal material, as otherwise 
designs may be disadvantageous for students with dyslexia, and educational access may be classified as a 
“wicked problem” (Mertens, 2015). In addition, multimedia research has paid little attention to images other than 
schematic pictures or diagrams (Scheiter, Wiebe, & Holsanova, 2008). Using replications of paintings from 
different artistic traditions, as in our study, may enrich earlier results in the field of multimedia. It would be 
valuable in the design of learning materials to know whether certain kinds of pictures are more likely to attract 
early inspection, create a holistic impression, and thus impact on reading comprehension. Further, with a mixed 
methods approach, more complex answers can be obtained, shedding light on how dyslexic readers process text 
and pictures that are integrated. 

A general assumption about the benefit of using pictures together with texts is that when the same theme is 
presented in two formats it is possible for the reader to make connections between the text and the picture and 
use those connections to reach a deeper level of comprehension. This presumes existing congruency between 
pieces of information, which is not always the case. Ariasi and Mason (2011) reported from an eye-tracking 
study that learners fixated longer on information that conflicted with their earlier conceptions, which is in line 
with the results Underwood and Foulsham (2006) presented about perceiving the gist of a non-congruent picture. 
Thus, our first hypothesis was about whether different motifs in the pictures mattered for time of inspection and 
dwell time on the picture. Our second hypothesis was related to promising learning effects from displaying a 
picture before textual information, obtained for example by Eitel et al. (2012). On the basis of this, we expected 
that learners would show a deeper level of comprehension if they inspected the pictures before reading, and thus 
had a holistic impression of the information to be processed (Hypothesis 2). Finally, with regard to the third 
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hypothesis, we wanted to establish at which points in the text participants chose to transition to the picture. We 
expected certain signal words to trigger the inspection of the picture as readers attempted to make connections 
between the text and the picture (Hypothesis 3). 

As a background to this study, we provide here a short description of the original experiment (Authors, 
submitted), from which new data was drawn for this re-analysis. The original experiment had 46 participants. All 
were screened for phonological ability with the DUVAN instrument (Lundberg & Wolff, 2003; Wolff & 
Lundberg, 2003) to ensure that participants could be divided into two groups, dyslexic and control. The 
experiment had two conditions: text only or text and picture, thus following a 2x2 design. Stimuli in the text only 
condition contained verbal information about six genres of art (abstract art, impressionism, cubism, romanticism, 
surrealism and pop-art) and in the text and picture condition an illustration was used as visual information in 
addition to the verbal. The texts in both conditions provided information about how motifs were composed, 
frequent motifs in that specific art genre, and the techniques and intentions of the artists in each genre. The 
pictures visualized information about motifs, composition, and techniques. The texts (an example i translated 
into English in the appendix) were presented on the left half of the screen and had a mean of 81 words. 

Stimuli were shown on a computer screen, and participants used the mouse to click through six screens with 
information and questions six times. The six screens comprised of: 

1. Written name of an art genre. 

2. Information in either text only or text and picture about the named genre 

3. An empty screen - research leader asked: “What information about [the art genre] did you get from 
the last screen?” 

4. Multiple-choice text question about the current genre with one correct answer. 

5. Ranking on Likert-scale how confident they were about the answer 

6. A pictorial question where participants choose among three paintings the one that was an example 
of named genre. 

The experiment ended with one question about how familiar the participant was on beforehand with the 
information in the experiment, rating from not at all to very familiar on a four point Likert-scale. 

METHODS 

Guetterman, Fetters and Creswell (2015) list three types of basic design in mixed methods studies: exploratory 
sequential, explanatory sequential and convergent designs. This study used convergent parallel mixed method, as 
both qualitative and quantitative data were collected at the same time. Both data sets were then analyzed in 
different ways, qualitatively and quantitatively, to answer the hypotheses. 

As a starting point, to find out if groups differed in inspection patterns, we compared eye-tracking data 
consisting of registered events, expressed as numbers. Specifically, we compared the means of number of 
fixations, dwell times on different areas of the stimuli, and average fixation time between groups and between 
types of pictures. The affordances built into the eye-tracking technology enabled us to consider the quantitative 
registrations of eye-movements as “footprints” of someone else’s inspection patterns. We seized this opportunity 
to look with another individual’s gaze; the recorded eye-movements over the stimuli created a scan path, similar 
to a movie. To find where in the text participants shifted from text to picture, we played the scan paths at 10% of 
normal speed twice and mapped transition points from text to picture. During this process we were forced to 
experience other people’s choices about where to look at a specific time. We have categorized this data set as 
quantitative and qualitative, due to the fact that we here use quantitative data that we interpret as expressions of 
consciousness. This particular method emerged during the process (Klassen et al., 2012) and was not planned 
beforehand. By the use of this method we could investigate whether a certain word or passage in the text resulted 
in transitions between text and picture (see Figure 7). The three texts were divided into 10 units, each 
representing 10% of the whole text. For example, the text about surrealism consisted of 90 words in Swedish, 
and the first unit was the first nine words of the text and so on. The units were used to measure and compare how 
far into each text participants read before they inspected the picture. 

We analyzed the oral answers with a qualitative approach to measure levels of reading comprehension. Answers 
were transcribed from the web-cam recordings, coded with a number, and read as a whole several times. The 
readings were aimed first at finding out how the answers related to the original text, and then whether 
participants used words directly connected to the text or paraphrased to create an independent explanation. 
Finally, we established whether the oral answers were merely repetitions of the texts or interpretations of what 
participants had read. Thus, both utterances and eye-movements were interpreted and integrated in order to 
investigate how patterns of transition between text and pictures differ or are similar between young adults with 
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or without dyslexia. See Figure 2 for a description of which types of analysis we used to investigate the 
hypothesis and in which order the different data sets were employed. 


How did groups' attend to 
the different pictures? 

(investigating H 1) 


When in time did 
particpants attend to 
pictures? 

(investigating H 3) 


How did participants 
understand the 
information? 

(investigating H 2) 



Figure 2. Types of data sets (in chronological order). 


Participants 

Participants’ (^=24) mean age was 23.2 years. Ten participants were previously diagnosed by professionals and 
had been found to have developmental dyslexia. The remaining 14 participants were assigned as a control group. 
The maximum score on the phonological test DUVAN was 294 points; the mean for the control group was 214 
points (SD =29 points) and for the group with dyslexia 123 points ( SD=15 points). Participants also rated their 
pre-knowledge about art on a five-point scale. The group with dyslexia had a mean of 2.5 (SD=1.3) and the 
control group had a mean of 2.6 (SD=IA) for self-rated pre-knowledge. 

Materials 

The text and the pictures representing three genres of art that were analyzed in this paper were chosen on the 
basis of variation in picture features, such as saliency or composition, between cubism, romanticism and 
surrealism. The cubist painting (see Figure 3) was more or less in one color, brown, although different shades 
and nuances of brown were used to create a motif. The picture was more a pattern than a depiction. It was clearly 
an abstract picture. 



Figure 3 Picture stimulus for cubism 

As a contrast, the painting from the romantic period (see Figure 4) had a clear and more realistic motif: a man 
standing on a cliff looking out over a vast waterfall, his back turned towards the viewer. The colors were more 
distinct than in the cubist painting. 
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Figure 4 Picture stimulus for romanticism 

Finally, the surrealistic painting (see Figure 5) differed in a third way: there were many objects to be seen and 
the colors were quite clear, but there was a lack of congruity between the objects and even within the objects 
themselves. The figure in the center might first be seen as an elephant, but a second look might reveal that it is 
no ordinary elephant. The picture shows real objects but in a distorted way, typical of a non-congruent picture as 
described by Underwood and Foulsham (2006). The non-congruency is related to violations of expected reality. 



Figure 5 Picture stimulus for surrealism 


Apparatus 

Eye-tracking data was collected via a SMI RED250 eye-tracker to record eye movements at a rate of 250 Hz via 
iView X software (v. 2.7.13). Stimuli were shown on a 22-inch Dell monitor with a resolution of 1680 x 1050 
and presented using Experimentation Center (v. 3.0. 155) software. 

Retention test 

During the experiment, one open-ended question was asked about the previously viewed material. Participants 
could then express freely what they had read and seen, thus showing their comprehension of the learning 
material. Participants’ answers were recorded with a web-cam. In this paper individual answers about the three 
pictures (cubism, romanticism and surrealism) are analyzed with respect to whether they were merely repeating 
the given information or whether they paraphrased it. A total of 72 answers have been categorized into either 
repeating or paraphrasing (3 genres x 24 participants = 72 answers). 

ANALYSIS AND RESULTS 

Eye-tracking measures 

To investigate whether different motifs mattered for inspection (Hypothesis 1), fixations from the two groups 
were first counted together. Pictures attracted different numbers of fixations, from 5.5 on the painting in the 
romantic style, to 6.1 on the cubist painting and about twice that (12.4) on the surrealist painting (see Table 1 for 
all measures). The dyslexic group’s average fixation length was higher on the surrealistic picture, approaching 
the average length of the control group’s fixations. 
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Table 1 Means and standard deviations of number of fixations and average length of fixations on each of the 
_ three pictures for the group with dyslexia and the control group. _ 


Condition 


Cubism 


Romanticism 

Surrealism 




Number 

of 

fixations 

Average 
fixation 
length (ms) 

Number 

of 

fixations 

Average 
fixation 
length (ms) 

Number 

of 

fixations 

Average 
fixation 
length (ms) 

Group with 
dyslexia 

Mean 

6.3 

119.7 

4.6 

119.8 

9.7 

213.6 

?z=10 

SD 

8.3 

110.4 

6.0 

132.5 

9.1 

94.4 

Control 

group 

Mean 

6.0 

220.1 

6.2 

262.4 

14.3 

230.7 

72=14 

SD 

4.3 

88.0 

3.8 

54.1 

9.9 

77.3 

Total 

Mean 

6.1 

178.3 

5.5 

203.0 

12.4 

223.6 


SD 

6.1 

108.2 

4.8 

117.0 

9.6 

83.2 


Because it has been proposed that the length of fixations is related to cognitive processing (Holmqvist et al., 
2011; Just & Carpenter, 1980), dwell times on the three pictures were of interest, and whether they differed 
between the two groups was also of interest. In order to assess the effect picture type had on dwell time for each 
group (Hypothesis 1), a multivariate analysis of variance, with dwell times on pictures as the dependent variable 
and condition (text or text and picture) as the independent variable, was performed. Results showed a main effect 
of between-subject factor (F (1,22) = 5.134, p < .03). The multivariate analysis also showed a within-subject 
factor (F (2,22) = 13.50, p < .001). To find out which picture caused this, a follow-up one-way ANOVA was 
performed, with dwell times on pictures as the dependent variable and condition as the independent variable. It 
showed a significant effect for the surrealistic picture (F (1,22) = 5.686, p = .026). Neither of the other two 
pictures had significant values for dwell times (cubist picture p =.24 and romantic picture p=. 065). As shown in 
the graph in Figure 6, the surrealistic picture was inspected for longer than the other pictures in both groups, but 
substantially longer in the control group. 



Figure 6 Plotted mean dwell times on pictures in percentage 


The fixation mapping revealed individualized ways of handling pictorial information, especially in the group 
with dyslexia. Four individuals from that group did not fixate at all on either the cubist or the romantic picture; 
however, three of the four inspected the surrealistic picture. The control group’s responses were more even: each 
one fixated at least once and the standard deviations were lower. 
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Interplay of text and picture 

Hypothesis 3 proposed that certain content-rich words might prompt participants to inspect the picture, but no 
such effect was found. At most only 4 participants out of 24 chose the same particular word as a transition point. 
This accords with the individualized processing of pictorial information as described earlier. However, the 
mapping of transitions between text and picture showed a difference in when inspection of the three pictures 
occurred. The diagram below (Figure 7) shows that most participants started looking at the screen about 
surrealism by inspecting the picture, while the picture of cubism had more fixations after participants had read 
the entire text. 



Figure 7 Diagram of transition points from text to picture 


Time spent on the text was longer for the dyslexic group, which not was surprising, as reading is more 
problematic. Figure 8 shows the mean dwell time in milliseconds on texts for each group. 


Dwelltime on texts 
Mean 



Group and text-stimuli 


Figure 8 Mean of dwell time in ms on text for each group 
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The first fixation on the screen for everyone in both groups was in the center, between the text and the picture. A 
flaw in the study design is the composition of the screen, which makes it impossible to avoid processing the 
picture in the peripheral visual field while reading the text. Therefore, participants knew after this first fixation 
on the screen that there was a picture to the right, but they probably also knew from their peripheral view that 
something was “wrong” with the surrealistic picture. Or, as explained by Underwood and Foulsham (2006), the 
composition of the picture violated the gist. The duration of the first fixation on pictures, which occurred after 
the initial fixation on the screen, had, for both groups counted together, a mean of 175 ms. The control group’s 
mean first fixation duration on the surrealistic picture was 205 ms, while for the dyslexic group it was 140 ms. 
This was the lowest mean fixation duration for the control group, but the highest for the dyslexic group. 

Dwell times and number of fixations showed that participants looked more often and longer at the non-congruent 
surrealist picture and less at the non-figurative cubist picture (see Table 1), even if that was also “different” from 
reality. Mapping the transitions, as shown above, indicated that the surrealist picture attracted early attention, 
while the cubist painting attracted attention more towards and at the end of the inspection time on that screen. 
From these findings, what can be said about the quality of the oral answers given for these two genres of art? 

Reading comprehension effects in relation to inspection patterns 

Participants expressed freely what information they had received from a screen that they had recently studied. In 
the analysis, the oral answers were not judged as either right or wrong. Furthermore, the utterances are treated as 
qualitative data. The aim with the analysis of the oral answers was to find out if the reading comprehension level 
seemed consistent over the three pictures or if it differed, and if it was in any way related to the inspection 
patterns of the pictures. One way to find out about the reading comprehension level was to look for independent 
reflections, where participants showed signs of having integrated what they read with previous knowledge, one 
mark of a deeper reading comprehension (Kintsch & Rawson, 2007). Participants who inspected the picture 
before reading the text were expected to boost their comprehension of the text (Hypothesis 2). The twenty-four 
participants gave answers on three art genres, producing a total of 72 answers. Each answer was, as mentioned 
above, categorized into one of the two categories repetitive or paraphrasing. The first and the second author 
independently ranked the oral answers into one of the two categories. Repetitive means here that the information 
from the previous screen was merely repeated, sometimes even verbatim. No new information or interpretation 
or inferences could be traced in this category. In the other category, paraphrasing, participants might give new 
information, present new examples, or use information from previous texts, pictures or pre-knowledge. 
Cronbach’s alpha for the 72 judged answers from the two raters was .82, an acceptable value of reliability. 

Disagreements were resolved by further discussions between the authors. 

The oral answers about cubism and surrealism, which differed most in dwell times and thus will be focused on 
further, showed that cubism was mainly described by merely repeating phrases from the text, sometimes even 
verbatim; the answers for surrealism were more likely to paraphrase, interpret, and add relevant assumptions to 
the information on the screen. 

Two examples from four different participants will illustrate 1/ how they each repeated what they had read on 
cubism, using almost the same words as the text, and 2/ in describing surrealism they interpreted what they had 
read and expressed ideas that went beyond the actual text. 

Example 1: 

Participant 1 with dyslexia, cubism : It started in the 20th century; the artists were inspired by African art; they 
painted humans in parts, and it was very colorless in the beginning and then colors returned. 

Participant 2 with dyslexia, cubism : It was an art form during the 20th century, where you took humans and 
landscapes and made it into geometrical parts. 

Participant 1 without dyslexia, cubism : It was the first abstract art, started in the 20th century; they depicted 
nature using geometrical shapes; they were inspired by African art; in the beginning it was colorless, but later it 
got more colorful. 

Participant 2 without dyslexia, cubism: This was late 1800s or early 1900s; it was inspired by African art, and it 
was about dividing paintings into geometrical figures, in the beginning the colors were monochrome, but that 
changed later on. 

All four participants mention when cubism began; three mention the source of its inspiration; and all four 
mentions both the separation of images into parts and the use of colors. The information in each case is given in 
the same order as it is given in the text. 
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Example 2: 

Participant 1 with dyslexia, surrealism : It started in France around 1920, and instead of having a realistic 
perspective they, the artist, let go of all his thoughts and used imagination, they could take things and, to give an 
example, a hot dog that is green. 

Participant 2 with dyslexia, surrealism: I found out that surrealism came from France around 1920; it was an art 
form where you twist reality, so to speak, and really distort it so you get all kinds of strange things, like that 
something that is as small as an ant can be as big as an elephant while the elephant is the size of the ant; it is 
about perspectives. 

Participant 1 without dyslexia, surrealism : It arose in France around 1920 and it is about passing the barriers of 
logic and continuing down, down to the unconscious; according to the text the artists somehow flipped things in 
their heads, and in one way it is about disregarding all the logical rules about how proportions of things are 
supposed to be; for example, a comb and something else, one erases the boundaries and things become 
surrealistic; they went behind reality, so to speak. 

Participant 2 without dyslexia, surrealism : It arose in France around 1920; it should be a little more dreamlike; it 
should not be based upon the everyday logic we are used to. For example, they took an egg, an egg could be the 
size of the sun or a green hot dog could be the same as a comb, it was as if the things floated together with 
distorted things, this made it hard to sort out what you saw. 

In these latter answers about surrealism all four participants mention the place and date of the origin of the genre, 
but their descriptions of the main ideas in surrealism are individual and they also chose different examples from 
the text to exemplify their reasoning. A compilation of the answers for cubism, romanticism and surrealism is 
shown in Table 2. 


Table 2 Type of oral answers on cubism, romanticism and surrealism 



Cubism 

Romanticism 

Surrealism 


Rep 

Para 

Rep 

Para 

Rep 

Para 

Dyslexic group 

10 

0 

10 

0 

7 

3 

Control group 

11 

3 

11 

3 

6 

8 


The answers for surrealism are more interpretative; participants paraphrase what they have read about surrealism 
more than they do for cubism. 

DISCUSSION 

With a traditional quantitative study, a plausible interpretation of the eye-tracking data might have led to the 
conclusion that non-dyslexic participants engaged more with the pictures. This is true, in one way, but as we 
have shown in the results section, there was more to understand around how the material was processed. For 
example, both groups paid more attention to the surrealistic picture, and both groups inspected it early on in the 
processing of that particular screen, which seemed to positively affect the reading comprehension. Moreover, the 
“following the gaze” analysis was crucial for understanding the qualitative differences between the oral answers. 
Thus, we could express the analysis as 1+1=3 (Fetters & Freshwater, 2015) that is, that the outcome of a mixed 
methods study is more than the sum of than its parts. In this paper the ambition was to reach a deeper 
understanding of how the material used in the study was experienced. The conclusions drawn from this 
integration of results and analysis would have been impossible to draw on the basis of one method. This is 
indeed a huge advantage of mixed methods research (Li, Marquart, & Zercher, 2000) and a challenge for eye¬ 
tracking research not to settle for merely quantitative data sets. 

Main findings 

The main aim of this paper was to find out whether different types of pictures gave the same transition pattern 
between text and picture, or whether the type of picture influenced the inspection. If the picture influenced 
inspection, were there any differences between the two groups? If a specific picture elicited more early 
inspection, could it be said to affect reading comprehension of that screen? Many participants inspected the 
surrealist picture before they actually began reading the text and this picture also attracted more fixations in total. 
This verifies Hypothesis 1, that the pictures themselves can matter for time-to-inspection and dwell times. A 
between-subjects main effect was found, showing that participants from the control group spent more time on the 
pictures than the dyslexic group. 

Reading ability thus seems to matter both to the number of fixations on each picture and to the associated dwell 
times. In the oral answers for surrealism, where the picture was inspected most often before reading, more 
participants expressed individualized interpretations of the learning material. Instead of merely repeating what 
they had read, they made inferences and used previous knowledge, as demonstrated by the fact that they inserted 
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information not found in the texts. This partially supports Hypothesis 2, that learners would show a deeper 
comprehension when pictures were inspected before reading material such as that used in this study, in which 
text and picture treat the same content. Finally, no words were found to have a trigger effect on transitions 
between text and picture. Thus Hypothesis 3 was rejected. 

Non-congruent motif attracts attention 

Clearly, the surrealistic picture was inspected the most often and the earliest. This means that readers were more 
likely to use the information from that picture to enrich the text. The control group had a greater number of 
fixations and longer dwell times on that picture, as well as longer fixations on the other pictures, than the group 
with dyslexia. Even so, the group with dyslexia benefited particularly from the surrealistic picture, as they too 
tended to inspect it before reading instead of at the end of the reading session. 

Why then is the surrealistic picture the most inspected? We claim, that it is the characteristic of the picture that is 
crucial, not the art genre. The transition patterns of the participants indicated that by perceiving the gist of the 
screen, participants discerned the non-congruence of the surrealistic picture. This seemed to provoke an impulse 
to inspect the image further before reading the text. We know from earlier research (Underwood & Foulsham, 
2007) that pictures with non-congruent objects attract the attention and give rise to longer fixations. The results 
of this study are in line with these findings. Thus, it can be assumed that participants decide from the gist they 
obtain from the screen whether or not to study the picture initially or at length. In this case the assumption is that 
the non-congruent surrealist picture attracted attention because it differed from participants’ pre-knowledge of 
what pictures normally look like. In variation theory (Marton & Booth, 1997; Marton & Pang, 2006), which is 
the framework for learning used for the design of the original experiment, (Authors, submitted), differences and 
contrasts are essential. It is when something differs from the background that individuals discern it. It is noticed 
as a contrast to a well-known background, thus the individual may discern what something is and become aware 
of it by distinguishing it from a background of what it is not (Ling & Marton, 2011). Without having discerned 
and been aware of a phenomenon or an aspect of a phenomenon, individuals cannot process it. 

Early inspection facilitates holistic processing of texts 

The early inspection of the surrealist picture may have had a positive impact on reading comprehension, as 
answers about this genre were more independent of the stimulus text. Regardless of whether the participants’ 
intention in inspecting the picture was to better understand the text, it seemed to have that effect. Because 
participants spent more time processing this picture, they were also able to make connections between what they 
saw and what they read. Pictures therefore do seem to have an impact on reading comprehension, while 
inspecting the screen on a global level, which is getting the gist, was not enough to improve comprehension. 
Something must trigger a more detailed inspection on the local level, thereby creating the possibility of 
constructing a situation model (Kintsch & Rawson, 2007). It seems only the non-congruent picture of surrealism 
provided that trigger: the majority of the participants inspected it, and neither of the other pictures, before 
reading the text. 

Furthermore, it seems that because reading is more of a cognitive effort for participants with dyslexia, they had a 
stronger urge than controls to start processing the text, and therefore disregarded illustrations more than the 
control group. They spent less time on pictures than the control group, whose total dwell times on pictures were 
significantly higher. The viewing behavior of the group with dyslexia fitted more closely with earlier results 
about the text-driven processing of multimedia materials (Hegarty & Just, 1993; Rayner et al., 2001; Schmidt- 
Weigand et al., 2010). Perhaps this is a wise strategy employed to save mental resources in order to be able to 
process written text, which is regarded as a higher order cognitive process than that for pictures (Rapp, Broek, 
McMaster, Kendeou, & Espin, 2007). On the other hand, when the surrealist picture was inspected before 
reading, the comprehension expressed in several oral answers approached that of a situation model (Kintsch & 
Rawson, 2007). Further, the inspection of the surrealist picture was especially beneficial for readers with 
dyslexia, since none of them produced an interpretive answer from the other two art genres (see Table 2). This is 
particularly interesting since the control group actually inspected the surrealistic picture more than the group 
with dyslexia, which highlights the importance of when a picture is fixated upon. It is thus not only the number 
of fixations or the dwell time that matters; it is also whether inspection occurs before reading. For the group with 
dyslexia this early inspection made a difference and it seems that it was a factor that contributed to the 
qualitative improvement of oral answers. 

CONCLUSIONS 

As we did not restrict our research to either a quantitative or a qualitative approach, we became able to find more 
complex answers to the research questions. We found from the fixation mappings that the participants perceived 
the gist of each screen during the initial centrally placed fixation. We can therefore conclude that a person does 
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not need to fixate on a picture in order to process it globally; the picture may very well be processed through 
peripheral vision during apprehension of the gist of a text-picture integrated medium such as the screens in the 
experiment. Fixations on areas of interest are thus not the only measure of how people process information. 
Getting the gist is an important first step in understanding a text, but not enough. Further comprehension seems 
to benefit from an early local inspection of the pictorial element. The progression from a global to a local level of 
inspection allows the possibility of constructing a situation model of the reading material. A major challenge for 
teachers and others involved in education is to improve the design of learning materials to ensure that readers are 
prompted to start inspecting pictures and illustrations on a local level, rather than settling for only the gist. 

ACKNOWLEDGMENTS 

This work was funded by The Swedish Research Council, grant number 721-2010-5379, under the project title 
“Double information, more or less comprehension.” 

Appendix 

REFERENCES 

Ariasi, N., & Mason, L. (2011). Uncovering the effect of text structure in learning from a science text: An eye¬ 
tracking study. Instructional science, 39(5), 581-601. 

Authors. (Submitted). Eye-tracking as an indicator of impaired executive function in people with dyslexia. 
Bazeley, P., & Kemp, L. (2012). Mosaics, triangles, and DNA metaphors for integrated analysis in mixed 
methods research. Journal of mixed methods research, 6(1), 55-72. 

Beacham, N. A., & Alty, J. L. (2006). An investigation into the effects that digital media can have on the 
learning outcomes of individuals who have dyslexia. Computers & Education, 47( 1), 74-93. 

Calvo, M. G., Nummenmaa, L., & Hyona, J. (2008). Emotional scenes in peripheral vision: selective orienting 
and gist processing, but not content identification. Emotion (Washington, DC), 5(1), 68. 

Eitel, A., Scheiter, K., Schuler, A., Nystrom, M., & Holmqvist, K. (2012, 28-31 August). How a Picture Can 

Scaffold Comprehension of Text. Paper presented at the Staging knowledge and experience: How to take 
advantage of representational technologies in education and training?, Grenoble, France. 

Engen, L., & Hoien, T. (2002). Phonological skills and reading comprehension. Reading and Writing, 15(1), 
613-631. 

Everatt, J., & Reid, G. (2009). Dyslexia: An overview of recent research. In G. Reid (Ed.), The Routledge 
Companion to Dyslexia (pp. 3-21). London: Routledge. 

Fetters, M. D., & Freshwater, D. (2015). The 1+ 1= 3 Integration Challenge. Journal of Mixed Methods 
Research, 9(2), 115-117. 

Greene, J. C. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of mixed methods 
research, 2(1), 7-22. 

Guardiola, J. (2001). The evolution of research on dyslexia. Anuario de Psicologla, 32(1), 330. 

Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in 
health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 
554-561. 

Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of 
Memory and Language, 32, 717-742. 

Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Halszka, J., & van de Weijer, J. (2011). Eye tracking: 

A comprehensive guide to methods and measures. Oxord: Oxford University Press. 

Houts, P. S., Doak, C. C., Doak, L. G., & Loscalzo, M. J. (2006). The role of pictures in improving health 
communication: a review of research on attention, comprehension, recall, and adherence. Patient 
education and counseling, 61(2), 173-190. 

Jewitt, C. (2015). Multimodal analysis. The Routledge Handbook of Language and Digital Communication, 69. 
Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological 
review, 87, 329-354. 

Kintsch, W., & Rawson, K. (2007). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of 
reading: A handbook (pp. 209-226). Malden, MA: Blackwell publishing. 

Klassen, A. C., Creswell, J., Clark, V. L. P., Smith, K. C., & Meissner, H. I. (2012). Best practices in mixed 
methods for quality of life research. Quality of Life Research, 27(3), 377-380. 

Laasonen, M., Lipsanen, J., & Virsu, V. (2012). Adult developmental dyslexia in a shallow orthography: Are 
there subgroups? Reading and Writing, 25(1), 71-108. 

Levesque, S., & Corrigan, J. (2014). What Does the Eye See? Reading Online Primary Source Photographs in 
History. Contemporary Issues in Technology and Teacher Education, 14(2), 101-140. 

Li, S., Marquart, J. M., & Zercher, C. (2000). Conceptual issues and analytic strategies in mixed-method studies 
of preschool inclusion. Journal of Early Intervention, 23(2), 116-132. 


Copyright © The Turkish Online Journal of Educational Technology 


112 




TO JET: The Turkish Online Journal of Educational Technology - January 2017, volume 16 issue 1 



Ling, L. M., & Marton, F. (2011). Towards a science of the art of teaching: Using variation theory as a guiding 
principle of pedagogical design. International Journal for Lesson and Learning Studies, 7(1), 7-22. 

Lundberg, I., & Wolff, U. (2003). DUVAN. Dyslexiscreening for ungdomar och vuxna. Stockholm: 
Psykologiforlaget. 

Marton, F., & Booth, S. (1997). Learning and awareness. Mahwah, N.J.: Lawrence Erlbaum. 

Marton, F., & Pang, M. F. (2006). On some necessary conditions of learning. The Journal of the Learning 
sciences, 75(2), 193-220. 

Mertens, D. M. (2015). Mixed methods and wicked problems. Journal of mixed methods research, 9(1), 3-6. 

Paulesu, E., Demonet, J. F., Fazio, F., McCrory, E., Chanoine, V., Brunswick, N.,. . . Frith, C. D. (2001). 
Dyslexia: cultural diversity and biological unity. Science, 297(5511), 2165. 

Peterson, R. L., & Pennington, B. F. (2015). Developmental Dyslexia. Annual review of clinical psychology, If 
283-307. 

Rapp, D. N., Broek, P. v. d., McMaster, K. L., Kendeou, P., & Espin, C. A. (2007). Higher-order comprehension 
processes in struggling readers: A perspective for research and intervention. Scientific Studies of reading, 
77(4), 289-312. doi:10.1080/10888430701530417 

Rayner, K., Rotello, C. M., Stewart, A. J., Keir, J., & Duffy, S. A. (2001). Integrating Text and Pictorial 
Information: Eye Movements When Looking at Print Advertisements* 1. Journal of Experimental 
Psychology: Applied, 7(3), 219-226. 

SaB, S., & Schiitte, K. (2015). Helping Poor Readers Demonstrate Their Science Competence Item 

Characteristics Supporting Text-Picture Integration. Journal of Psycho educational Assessment , 
0734282915588389. 

Scheiter, K., Wiebe, E., & Holsanova, J. (2008). Theoretical and instructional aspects of learning with 
visualizations. Cognitive effects of multimedia learning , 67-88. 

Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). A closer look at split visual attention in system-and 
self-paced instruction in multimedia learning. Learning and instruction, 29(2), 100-110. 

Snowling, M. (2000). Dyslexia : Wiley-Blackwell. 

Underwood, G. (2005). Eye fixations on pictures of natural scenes: Getting the gist and identifying components. 
In G. Underwood (Ed.), Cognitive processes in eye guidance. Oxford: Oxford University Press. 

Underwood, G., & Foulsham, T. (2006). Visual saliency and semantic incongruency influence eye movements 
when inspecting pictures. The Quarterly Journal of Experimental Psychology, 59(11), 1931-1949. 

Wolff, U., & Lundberg, I. (2003). A Technique for Group Screening of Dyslexia among Adults. Annals of 
Dyslexia, 53,324-339. 


Copyright © The Turkish Online Journal of Educational Technology 


113 




TO JET: The Turkish Online Journal of Educational Technology - January 2017, volume 16 issue 1 



Surrealism emerged in France around 1920. It has 
been explained by die word dream-pictures. In a 
surrealistic painting style artists were supposed to 
let go of logical thinking and let unc onscious and 
unrealistic combinations of things meld. By 
pushing reason aside the artist released barriers that 
made an egg look smaller than the sun or that a 
table has an even surface. Instead a connection 
between for example a green hot dog and a gigantic 
comb were necessary. The surrealists tried to depict 
the world without rules for how to paint it 
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